Deliverable_11_14

Low-Rank Kalman Filter

Our main goal for this week was to apply Jovo’s low-rank Kalman filter to a dataset and check its effect on discriminability. We spent a good amount of time going through each of the provided code files and understanding what each one was doing. We then moved on to replicating the experiments. We began by first running the parameter estimation and KFS algorithm code on simulated data, as per the specifications in the code package. Next, we tested these algorithms on the given real data using the provided scripts. The resulting plots were the same as earlier testing found here.

Finally, we modified some R and MATLAB code to be able to run Jovo’s algorithms on an actual full dataset:

require(R.matlab)
source('C:/Users/Eric/Documents/Github/fngs/data_processing/Rutils/open_timeseries.R')

fnames = list.files('2_JHU/Research/Data/BNU1', pattern="\\.rds", full.names=TRUE)
parsed = open_timeseries(fnames, sub_pos=3, run_pos=4)
ids = parsed[[3]]
ts = parsed[[1]]
nSubs = length(ids)
nRois = dim(ts[[1]])[2]
nStps = dim(ts[[1]])[1]-1
wgraphs = array(dim=c(nStps, nRois, nSubs))
for(i in 1:nSubs) {wgraphs[,,i] <- ts[[i]][1:nStps, 1:nRois]}

for(i in 1:nSubs) {
  writeMat(con=paste('2_JHU/Research/Data/BNU1/timeseriesdata-',i,'.mat',sep=''),y=wgraphs[,,i],n=nRois)
}

for i = 1:106
load(strcat('Data/BNU1/timeseriesdata-',int2str(i),'.mat'));

y = y';
[a,c,q,r,pi,v] = kfs_learn(y,eye(n),eye(n),eye(n),eye(n),y(:,1),eye(n),1e-6,20);
[Fv1,Fv2,Fx1,Fx2,Sx,Sv,Scov] = KFS(a,c,q,r,pi,v,y);

save(strcat('Data/BNU1/KFS/filtered-',int2str(i),'.mat'));
end

require("ggplot2")
require("reshape2")
source('C:/Users/Eric/Documents/GitHub/Reliability/Code/FlashRupdated/functions/distance.R')
source('C:/Users/Eric/Documents/GitHub/Reliability/Code/FlashRupdated/functions/reliability.R')
source('C:/Users/Eric/Documents/GitHub/Reliability/Code/R/processing/hell_dist.R')
source('C:/Users/Eric/Documents/R/prod_pkde.R')
source('C:/Users/Eric/Documents/R/multiplot.R')

vars = vector("list", nSubs)
for (i in 1:nSubs) {
  vars[[i]] = readMat(paste('2_JHU/Research/Data/BNU1/KFS/filtered-',i,'.mat',sep=''))
}

c = array(dim=c(nRois, nRois, nSubs))
for (i in 1:nSubs) {c[,,i] = cor(t(vars[[i]]$Sx))}
dist = distance(c)
m = mnr(rdf(dist, ids))
pdistc_kf <- ggplot(melt(dist), aes(x=Var2, y=Var1, fill=value)) + geom_tile(color="white") +
  scale_fill_gradientn(colours=c("darkblue","blue","purple","green","yellow"),name="Distance") + labs(x="Scan", y="Scan", title=paste("MNR = ", round(m, digits=4)))
pkdec_kf = prod_pkde(dist, ids)
multiplot(pdistc_kf, pkdec_kf, cols=2)

Results for the BNU dataset:

BNU_1

BNU_1 Kalman Filtered

BNU_1 Kalman Smoothed

For the NKI and SWU datasets, several of the subject files encountered singular matrix errors during the KFS EM. This led to bad results for those scans, signified by the 0 distance in the discriminability plots.

Results for NKI:

NKI

NKI Kalman Filtered

Results for SWU:

SWU_1

SWU_1 Kalman Filtered

Conclusions:

It appears as though the kalman filtering did not improve discriminability. For the BNU dataset, we first estimated the parameters using EM with 20 iterations. The discriminability after filtering with these parameters was actually lower than it was before filtering.